Papers with long-context inference
LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can process extremely long contexts, requiring efficient inference over extended inputs. |
| Approach: | They propose a model that uses a constant-sized key-value cache to train long-context models. |
| Outcome: | Experimental results show that LongSpec achieves 3.26x speedup over strong Flash Attention baselines and 2.34x wall clock time on four math reasoning tasks. |
UNComp: Can Matrix Entropy Uncover Sparsity? — A Compressor Design from an Uncertainty-Aware Perspective (2025.emnlp-main)
Copied to clipboard
Jing Xiong, Jianghan Shen, Fanghua Ye, Chaofan Tao, Zhongwei Wan, Jianqiao Lu, Xun Wu, Chuanyang Zheng, Zhijiang Guo, Min Yang, Lingpeng Kong, Ngai Wong
| Challenge: | Deploying large language models (LLMs) for long-context inference remains challenging due to their substantial memory and computational demands. |
| Approach: | They propose an uncertainty-aware framework that leverages truncated matrix entropy to identify areas of low information content. |
| Outcome: | The proposed framework reduces the KV cache size to 4.74% of the original and achieves a 6% speedup. |
S2O: Early Stopping for Sparse Attention via Online Permutation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing block-granularity sparsification can reduce latency, but coarse blocks impose an intrinsic sparsity ceiling. |
| Approach: | They propose a method that performs early stopping for sparse attention via online permutation. |
| Outcome: | The proposed approach reduces the complexity of the model and its performance. |
Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models face quadratic computational costs, information forgetting, and context fragmentation . current methodologies diverge into three paradigms, each facing inherent trade-offs between Equal contribution. |
| Approach: | They propose a framework that enables efficient long-context inference via chunk-wise compression and selective memory recall. |
| Outcome: | The proposed framework reduces peak GPU memory usage and speeds up inference on multi-hop reasoning benchmarks. |
ContrastKV: Robust KV Cache Eviction via Contrastive Signal Fusion for Multi-Query Generalization (2026.acl-long)
Copied to clipboard
| Challenge: | Existing query-agnostic approaches rely on a single proxy query, leading to fragile eviction decisions under high evict ratios. |
| Approach: | They propose a query-agnostic KV cache eviction algorithm that exploits complementary semantic and non-semantic signals. |
| Outcome: | Experiments show that the proposed algorithm outperforms state-of-the-art methods while retaining up to 92% accuracy with only 20% of the KV cache budget. |
APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs (2025.acl-long)
Copied to clipboard
Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Sun Ao, Hao Zhou, Jie Zhou, Zhiyuan Liu, Maosong Sun
| Challenge: | Long-context inference is crucial for advancing large language models, but its prefill speed remains a bottleneck. |
| Approach: | They propose an efficient long-context inference framework that leverages multi-host approximate attention to enhance prefill speed. |
| Outcome: | The proposed framework achieves speedups of 9.2, 4.2, and 1.6 without any degradation in performance. |
Evolving Sparsity: Leveraging Token Importance Dynamics for Efficient LLM Decoding with Sparse Attention (2026.acl-long)
Copied to clipboard
| Challenge: | Efficient long-context inference remains a major challenge for large language models (LLMs), as the cost of attention computation during auto-regressive decoding grows linearly with the context length. |
| Approach: | They propose to model token importance as a dynamic process that evolves over decoding steps and propagates through model layers. |
| Outcome: | The proposed method outperforms baseline sparse attention methods and achieves speedups of up to 5.36 for attention latency and 2.33 for end-to-end decoding. |
MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to optimize large language models for long-context inference are inefficient and consume memory. |
| Approach: | They propose a mixed-precision quantization method via mixture of experts that inputs tokens into router chunk by chunk to reduce inference overhead. |
| Outcome: | The proposed method outperforms state-of-the-art KV cache quantization methods on multiple benchmark datasets. |
StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing compression approaches prioritize tokens based on local saliency metrics to decouple prefill computation from decoding memory. |
| Approach: | They propose a structure-aware KV cache compression framework that prioritizes tokens based on local saliency metrics to decouple prefill computation from decoding memory. |
| Outcome: | The proposed framework preserves long-range dependencies and retrieval robustness. |
MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference (2025.acl-long)
Copied to clipboard
Kunxi Li, Zhonghua Jiang, Zhouzhou Shen, ZhaodeWang ZhaodeWang, Chengfei Lv, Shengyu Zhang, Fan Wu, Fei Wu
| Challenge: | Existing KV cache eviction methods fail to capture modality-specific information, resulting in suboptimal performance. |
| Approach: | They propose a modality-adaptive key-value (KV) cache eviction strategy to enhance the efficiency of multimodal large language models in long-context inference. |
| Outcome: | The proposed method reduces the KV cache memory footprint and model inference latency while maintaining high accuracy across multimodal long-context tasks. |
GNN-RAG: Graph Neural Retrieval for Efficient Large Language Model Reasoning on Knowledge Graphs (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to retrieval-augmented generation (RAG) rely on costly LLM calls to generate relation paths or traverse the KG. |
| Approach: | They propose a framework that uses lightweight Graph Neural Networks to enhance retrieval. |
| Outcome: | The proposed framework outperforms existing methods on multi-hop and multi-entity questions. |
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Rapid advances in Large Language Models have spurred demand for processing extended context sequences . however, performance degradation due to sequence lengths out-of-distribution and excessively long inference times are limiting LLMs in long-context scenarios. |
| Approach: | They propose a training-free method for efficient and accurate long-context inference . they selectively involves a few critical KV cache tokens in attention calculation . |
| Outcome: | The proposed method speeds up attention computation and accelerates inference time while reducing selection overhead. |
When Efficiency Meets Safety: A Benchmark Security Analysis of KV Cache Compression in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Key-Value (KV) caching is widely used in large language models to enable long-context inference efficiently, yet its security implications remain underexplored. |
| Approach: | They propose a history-aware, per-head feedback merging strategy that prevents safety degradation while maintaining efficiency. |
| Outcome: | The proposed strategy prevents safety degradation while maintaining efficiency. |
Squeezed Attention: Accelerating Long Context Length LLM Inference (2025.acl-long)
Copied to clipboard
Coleman Richard Charles Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, Sebastian Zhao, June Paik, Michael W. Mahoney, Kurt Keutzer, Amir Gholami
| Challenge: | Emerging Large Language Models require long input context to perform complex tasks. |
| Approach: | They propose an algorithm to reduce the complexity of attention with respect to the fixed context length. |
| Outcome: | The proposed method reduces the complexity of attention from linear to logarithmic with respect to the fixed context length. |
LeanK: Learnable K Cache Channel Pruning for Efficient Decoding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing efforts to optimize the key-value (KV) cache include: (1) Eviction, which discards cache of less important tokens; (2) Selection, which retains the full KV cache but selectively reads relevant entries. |
| Approach: | They propose a learning-based method that prunes unimportant key (K) cache channels by leveraging static channel sparsity. |
| Outcome: | Experiments show that LeanK reduces GPU memory and accelerates decoding without sacrificing accuracy. |